Informática Multimedia 1995 April

home *** CD-ROM | disk | FTP | other *** search

/ Informática Multimedia 1995 April / Informatica Multimedia CD - Epimundo.iso / DOS / FILEFIND / SNR.ZIP / SNR503.DOC < prev next >

Wrap

Text File | 1993-05-30 | 66.0 KB | 1,799 lines

TITLE: SNR.EXE VERSION: 5.03 PURPOSE: Multi-string simultaneous Search 'N' Replace program RELEASE DATE: 5/31/93 AUTHOR: THOMAS A. LUNDIN 16267 Hudson Avenue Lakeville, MN 55044 day phone: (612) 588-7571 nights/weekends: (612) 431-5805 CompuServe ID [70523,262] REGISTRATION: $50 per copy; volume and site licenses available. Registered version discards the opening shareware compliance screen, and a free multi-window DOS text editor is bundled with the package. Registration guarantees technical support; unregistered users will receive support on a time-available basis. Fill out and return the ORDER.FRM file along with your payment to register your copy. DESCRIPTION: SNR is a multi-string search-and-replace filter. Both text and binary files can be processed by the program. SNR will translate a file of any size that your system can handle. Version 5 of SNR brings many new advanced conversion capabilities to the program, such as: - wild-card conversion patterns - variable-length conversion patterns - start-of-file and end-of-file conversions - text field padding - table chaining - enhanced context flags Up to 2,500 multi-character (m:n) equations can be entered into an SNR table, each of which can be a maximum of 4,999 characters in length. Total conversion space is limited to 64,000 characters. All search-and- replace operations are performed in a single pass. Context flag conversion allows toggling between two different output strings for the same input string. Wild card patterns allow unspecific character strings to be matched and converted. -1- SNR+ VER 5.03 REVISION HISTORY 5.02 - 5/26/93 Major rewrite. Added advanced conversion features such as wild card pattern matching, variable-length patterns, field padding, and so on. Allow file names to contain DOS wild cards, also. Allow multiple-table chaining on the command line. Allow output file subdirectory specification. Greatly expanded program capacities. Allow 16 context flags. Change default bit-masking from 7-level to 8-level as this is the more harmless option. 4.0 - 7/19/90 Removed OS/2 compatibility. Nobody used it. Allow 256 1:1 equations to be loaded in addition to 50 m:n equations. Added context flag. 3.01 - 11/3/89 Program recompiled for family mode, meaning it will run as-is under DOS, OS/2 real mode, and OS/2 protected mode. Also improved I/O throughput on fast disk drives. 3.0 - 9/27/89 The program has been completely rewritten. The pesky bugs that were present in versions 1.5 and 2.0 have been eliminated. All binary characters INCLUDING NULLS can now be searched and replaced. Nulls can also be entered as part of a larger search- and-replace equation. An automatic "bit-stripping" feature has been added which removes the high-order bits from input characters, forcing 8-bit codes into 7-bit ASCII. The feature can be selectively disabled for binary files. The equation length has been increased to 200 characters, which can be split into search and replace sides of any length, as long as the total does not exceed 200. Equations no longer need to be in any specific order. The average speed of the program has been vastly improved, especially for multiple and high-occurrence search and replaces. The program no longer needs to read itself for runtime parameters. -2- SNR+ VER 5.03 PROGRAM OPERATION The command line invocation is: snr [@]filename ext table[ table2...] [/d] [/opathname] [/r] 1. "filename" is the name of the DOS file you wish to convert. The DOS wildcard characters "*" and "?" are allowed here, and they will result in all files matching the name pattern to be converted at one time. An at-sign (@) in front of the filename indicates that the file contains a list of file names to convert. There are two ways of creating the list of file names: the first is to use the DOS DIR command and redirect its output to a file, like this: dir *.txt >dirfile This will create a disk file named "dirfile" which contains a directory list of all files with a ".TXT" extension. You can edit this file if you want, to weed out certain files you don't want and retain the others. You can also create a list file manually by just creating a file with a text editor and typing in the path and file names for each of the files you wish converted. Example: d:\data\test.txt c:\bigfiles\tapedat.asc c:\windows\win.ini The converted output files will appear in your current directory. You can optionally direct the output to another directory with the /o option, explained later. 2. "ext" is the filename extension you wish to assign each output file. The output files are created by adding this extension onto the name of the input file. If you are converting multiple input files from a DIRLIST, or from a wild card in the command line, all of the resulting output files will have the same extension. The choice of an extension is important; if you accidentally choose an extension which is already used by another file, you will overwrite the existing file, losing its data in the process. SNR will not warn you of impending overwrites. Use an extension which you are sure is unique. Extensions can be the DOS device names CON, NUL, AUX, and PRN. In any of these four cases, an output file is NOT created on disk; rather, the output is redirected to the console (CON), nowhere (NUL), the rs232 port (AUX), or the printer (PRN). Using CON is handy for a quick preview of the conversion process before storing it to disk, although if you preview binary files to CON, be aware that hex code 1A (DOS end-of-file) will terminate the conversion display, perhaps prematurely. (This permature termination will NOT occur when you store to disk -- it is an artifact of displaying to the console.) Also, some binary characters will set the PC speaker beeping when you display to the screen. -3- SNR+ VER 5.03 3. "tablename" is the file name and optional path which contains your string translation "equations". Although no restrictions are placed upon the tablename (aside from conforming to the DOS naming format), for sake of clarity it is suggested that you adopt a consistent naming scheme for them, (say, with an *.S extension). Creating conversion equations is discussed in more detail later. You may specify more than one table at a time on the command line; the effect of this is that SNR will convert your input files through each table in sequence. The contents of the output files will be just as if you had run SNR a number of times in a row. 4. "/d" is an option which replaces the original input file with the converted data. The extension which you specify on the command line is used as a temporary intermediate file until the conversion is complete, at which time the program deletes the original input file and renames the temporary output file to the input name. 5. "/o" is an option which allows you to place the output files in some directory other than the current directory. The /o is followed immediately by the name of the directory path (no spaces in between). 6. "/r" is an option used only in conjuction with the /o option; it is meaningless by itself. /r forces all of the output files in the other directory to retain the same name as the original input files. At the end of the conversion run using this option, you will have two directories of files with the same names, but the data in the files in one of the directories will have been converted. 7. The options /o, /r, and /d may appear in a list file, after the name of the file to convert. Example: d:\data\file.1 /od:\ d:\data\file.2 c:\text\may.doc In the above example, all of the files in the list will be converted and stored on the root directory of drive D:. These option codes may NOT appear in a list file created by redirecting the DOS DIR command to a file. -4- SNR+ VER 5.03 COMMAND LINE EXAMPLES C>snr tstfil.doc txt tst.s The above command line will convert input file "tstfil.doc" through table "tst.s" and create output file "tstfil.txt". C>snr @dirlist p1 sample.s The above command line will convert a group of files listed in the file "dirlist" through table "sample.s" and create a group of output files with extensions of *.p1. C>snr *.c xxx pass1.s pass2.s pass3.s /d The above command line will convert a group of files matching the DOS file name pattern "*.c" through the three tables "pass1.s", "pass2.s" and "pass3.s" in order, and the converted output files will replace the original input files. (The temporary intermediate files will have an extension of *.xxx.) C>snr mytest.dat da2 pass1.s /od:\ The above command will convert the input file "mytest.dat" through table "pass1.s" and create an outfile named "mytest.da2" in the root directory of drive D:. -5- SNR+ VER 5.03 CREATING CONVERSION TABLES SNR tables are ASCII text files which contain the search-and-replace equations used by the program. Any word processor or text editing program capable of loading and saving plain ASCII text files will be sufficient to create conversion tables. (The registered version of SNR is bundled with a free multi-window DOS text editor for this purpose.) Up to 2,500 of these equations can be entered in a single table, and each equation can consist of up to 4,999 characters split freely between the search side and the replacement side. Blank lines in a table will be ignored. The maximum size of a table is limited to 64,000 characters. These equations will be matched against the input file in a single pass, resulting in a converted output file. EQUATION TYPES This manual may refer to certain equation types from time to time. The equation types that are handled by SNR are as follows: 1:0 an equation comprising one character on the left side and no characters on the right side 1:1 an equation with one character on the left side and one character on the right side 1:n an equation with on character on the left side and two or more characters on the right side m:0 an equation with two or more characters on the left side, and no characters on the right side m:1 an equation with two or more characters on the left side, and one character on the right side m:n an equation with two or more characters on the left side, and two or more characters on the right side FORMAT OF EQUATIONS The simplest form of conversion equation consists of literal search and replace text separated by an equals sign. A sample 1:1 equation would be: A=a The above equation would translate an upper-case 'A' to a lower-case 'a'. A sample m:n equation would be: Now is the time=NOW IS THE TIME The above equation will translate the words "Now is the time" to all upper-case. Notice that spaces ARE SIGNIFICANT characters in an equation, so don't use them carelessly if you don't mean to have them converted. 1:0 and m:0 equations ignore search strings (that is, throw it away on output). They are defined simply by leaving out a replacement string, like this: Now is the time= -6- SNR+ VER 5.03 If you want to output word spaces at the end of a replacement string, but your text editor strips trailing spaces, you can define them as hex codes: Now is the time =NOW IS THE TIME\20 In fact, any hex code can be formed from a backslash followed by two hex digits (0-9, a-f, A-F). You'd normally use hex codes to search or replace binary characters that can't be generated directly from the keyboard. For instance, a carriage return/line feed sequence (CRLF) can be specified like this: \0d\0a\0d\0a=\0d\0a The above equation will convert two CRLFs in a row to a single CRLF. SNR RESERVED CHARACTERS There are three ASCII characters which MUST be specified as hex codes in an SNR equation, since they have special meaning in their normal ASCII form. They are: Backslash (\), which must be entered as \5c Equals (=), which must be entered as \3d Asterisk (*), which must be entered as \2a If you forget to follow this notation, you will very likely trigger error messages when you attempt to run a conversion. EXTRA-LONG EQUATIONS SNR can handle equation lengths up to 4,999 characters. It's quite likely that you will never encounter this limit. Even so, your word processor or text editor may have a much smaller limit on the number of characters you can enter per line. Should you find you need to carry an equation over from one line to the next, you can end a line with \+ as the last item in the line, and SNR will automatically add the next following line to the same equation. You can have as many continuation lines as you need to. Example: Now is the time for all good men=NOW IS THE TIME \+ \20FOR ALL GOOD MEN \ This is all one continued equation Be mindful that any spaces that occur before the \+ continuation code are ignored (an exception to the all-spaces-are-significant rule). Any text that appears on the same line after the \+ code is ignored. ENDING YOUR TABLES The end of a table is signified by the end of your equations. You may also place a \\E on a line by itself as an end-of-table marker. This code is optional, but recommended, since it will prevent SNR from inadvertently reading past the intended last line of your equations (some word processors may pad their last blocks with garbage or with an EOF character, which SNR would attempt to read as the start of a new equation). A sure bet that you need to place the \\E code at the end of your tables is if SNR occasionally aborts with the message: Incomplete equation (missing right half): -7- SNR+ VER 5.03 ...and you are unable to see that any of your equations is incomplete. REPEATING CHARACTERS SNR provides a shorthand notation for describing a large number of repeating characters. Rather than entering an equation that looks like this: AAAAAAAAAAAAAAAAAAAAAAAAA=bbbbbbbbbbbbbbbbbbbb ...you can describe it like this: *(25)A=*(20)b Any character can be repeated using this notation. By changing the repeat value in parentheses, you accomplish the same thing as typing the repeat character that number of times. NOTE: SNR will actually expand a shorthand notation into longhand notation before it begins conversion, so make sure you don't inadvertently exceed the 4,999- character equation limit with your repeat values. COMMENTING YOUR EQUATIONS SNR provides for writing comments in your tables. Comments can be entered in a table as lines by themselves, or set off to the right of an equation, as in this example: \ This is a comment line by itself. \ A comment consists of a single backslash \ followed by one or more spaces. \0d\0a=\0d\0a \ this will ensure that existing CRLF pairs are \ left untouched \0d=\0d\0a \ this equation will convert an isolated CR \ into a CRLF \0a=\0d\0a \ this equation will convert an isolated LF \ into a CRLF \\E When a comment follows an equation on the same line, any spaces that occur before the comment are ignored; this is an exception to the all- spaces-are-significant rule. If you're a programmer, or you've used a macro language of some kind, you already know the value of program comments. If you're new to the field of user programming, you should immediately get into the habit of commenting your work. Believe me, you'll be glad you did, because a time will come when you have to make a change to a table you created months past, and you'll be left scratching your head over it if you haven't commented the equations. EQUATION ORDERING SNR will automatically sort equations by length when assembling a table in memory. SNR does not process equations in sequential order when it converts a file, so the order in which you enter your equations is largely immaterial. The exceptions to this rule are when using wild card patterns in your equations. This will be explained further on. -8- SNR+ VER 5.03 Equation processing is completed in a non-recursive fashion. That is, once an equation has been matched and a replacement made, the replaced data is not sent back for reconversion through another equation. It is passed onto the output file once. -9- SNR+ VER 5.03 CONTEXT FLAGS There are probably instances where you'd like to toggle between two different replacement strings for the same search string. The context flag allows you to do this. Some example uses would be: - to compress multiple spaces from a document - to ignore any text between two codes - to make one code alternate as two different codes - to turn all-upper-case text into upper and lower There are undoubtedly many other uses. There are 16 context flags which can be used, and each context flag has two states: on and off. The context flags are identified by the numbers 0-9 and the letters a-f (for a total of 16 individual flags). When SNR begins execution, the context flags' states are all OFF. A flag's state can be tested in a search, and its state can be set or reset in a replacement. A context flag is represented in an equation by an asterisk, followed by the number or letter of the flag, followed by a one or zero (for ON or OFF). A context flag is the last item entered in a search or replacement string. For example: ABC*00=abc*01 ABC*01=XYZ*00 In the above two equations, if the string "ABC" is read from the input file, and context flag 0 is OFF, then the string "abc" is written to the output file, and context flag 0 is set ON. If, on the other hand, the string "ABC" is read from the input file, and context flag 0 is ON, then the string "XYZ" is written to the output file, and context flag 0 is reset OFF. For example, if data from the input file looks like this: ABCDEFG ABCDEFG ABCDEFG ABCDEFG ...our small example table would convert it to this: abcDEFG XYZDEFG abcDEFG XYZDEFG Context flags are global in scope; this means that each flag is accessible by many different equations in one table, and any equation can test or set the flags. But beware of unwanted side effects -- remember that only one equation at a time will control the state of a flag, and its state may change between two equations that complement each other, resulting in misconversion. For example: ABC*00=abc*01 ABC*01=XYZ*00 EFG*00=efg*01 EFG*01=ZYX*00 -10- SNR+ VER 5.03 The above equations are similar to the previous example, except that we have defined two new equations that test and set a single context flag. So, given this example input data: ABCDEFG ABCDEFG ABCDEFG ABCDEFG ...the result would convert to this: abcDZYX abcDZYX abcDZYX abcDZYX The context flag is being toggled by only two equations: ABC*00=abc*01 <--- this one ABC*01=XYZ*00 EFG*00=efg*01 EFG*01=ZYX*00 <--- and this one If, on the other hand, the input data stream looked like this: ABCDEFG DEFGABC ABCDEFG DEFGABC ...the result would look like this: abcDZYX DefgXYZ abcDZYX DefgXYZ This conflict between two sets of equations grappling over the same context flag can be alleviated by using a different flag number for one of the sets: ABC*00=abc*01 ABC*01=XYZ*00 EFG*10=efg*11 EFG*11=ZYX*10 Now the equations will have independent flags, and the data stream will be processed using each equation separately. Now, this data stream: ABCDEFG ABCDEFG ABCDEFG ABCDEFG ...will be converted to this: abcDefg XYZDZYX abcDefg XYZDZYX The thing to keep in mind if you are going to use a context flag is: it is OFF when the program begins; you need to enter an equation to set it ON; and then you need to have some equation that resets it OFF. See some of the sample tables for examples in the use of context flags. -11- SNR+ VER 5.03 SPECIAL CONTEXT FLAG CODES Certain types of conversions involving context flags can be very tedious to define. Take the case of a table whose task is to compress two or more adjacent blank spaces into a single space. Using our context flags, we would start out by setting a flag when we first encounter a space: \20*00=\20*01 This equation states that if a space is encountered and flag 0 is OFF, it will be converted to one space and flag 0 ON. Next we define what happens to a space when flag 0 is ON: \20*01= This equation states that a space and flag 0 ON, will be ignored. That takes care of removing adjacent spaces, but there's something missing. The way we have it set up now, EVERY subsequent space will be removed, even between words, because we have not defined the case where flag 0 ever gets set OFF. Without something to set flag 0 OFF, only the second equation will hold true. Since we have defined the case where adjacent spaces will be ignored, we need to define the case where some character other than a space will reset flag 0 so we can retain the single interword space mentioned earlier. Here's where the tedium comes in. The complete solution is to make every character EXCEPT a space set flag 0 OFF, like this: \00=\00*00 \01=\01*00 \02=\02*00 \03=\03*00 \04=\04*00 \05=\05*00 ... \7c=\00*00 \7d=\00*00 \7e=\00*00 ... ...and so on. Basically, you'd need 255 similar equations to finish the task. Specifically for these types of conversions, SNR provides two specialty context flag codes: *ic and *ig. IGNORE CONSECUTIVE The Ignore Consecutive (*ic) flag code is used on the right-hand side of the conversion equation. Its purpose is to allow you to specify any set of characters that will be ignored when encountered adjacently in a data stream. A side benefit of this notation is that SNR will automatically generate all of the other flag reset equations that are needed to complete the conversion. This vastly reduces the "clutter" -12- SNR+ VER 5.03 that would otherwise be present in your table. For instance, here is our complete space compression table with the help of the *ic code: \20=\20*ic\20 That's all there is to it. This equation reads: one space equals one space, but ignore consecutive subsequent spaces. You may define more than one character after the *ic code. In fact, you may define as many characters after the *ic code as you care to. Each defined character will be ignored if they occur consecutively in the data stream after matching the string on the left side of the equation. Here's an example: .\20.\20.=<ldr>*ic\20. This equation will convert three periods separated by spaces into the string <ldr>, and ignore any subsequent consecutive periods and spaces. Thus, given the following input data: Price. . . . . . . . . . . . . . . $25.00 ...the equation would convert it to this: Price<ldr>$25.00 IGNORE GENERAL The other specialty context flag code is the Ignore General (*ig) code. This code appears on the right-hand side of an equation and its purpose is to cause all characters occurring after the equation to be ignored, until an Ignore Cancel (*ix) flag code is encountered. Example: [=*ig ]=*ix These two equations will cause anything falling between a left and right square bracket to be ignored, including the brackets themselves. Other characters falling outside of these delimiter characters will be converted in normal fashion. The *ig/*ix notation is the quickest way to remove enclosed groups of characters from a data stream. -13- SNR+ VER 5.03 START-OF-FILE AND END-OF-FILE CONVERSIONS SNR provides for special conversions to occur before and after normal conversion of a data stream. These conversions can be useful in adding specific headers or trailers to a file, or in removing portions of the beginning and end of a file. START-OF-FILE CODE To trigger an extra conversion at the start of the file, use the \s start-of-file code. \s can be entered only as the first character on the left-hand side of an equation. It may be used alone, or be followed by other characters. Example: \s=<BEGIN>\0d\0a The above equation will output the string <BEGIN> as the first item in a converted file. \s=*ic\20 The above equation will ignore any group of consecutive spaces occurring at the beginning of a file. \sNow is the time= The above equation will ignore the words "Now is the time" when they occur at the beginning of a file. END-OF-FILE CODE The \q end-of-file code works similarly to the start-of-file code. When it is used in an equation, it will trigger a conversion at the very end of a file. \q must always stand by itself on the left-hand side of an equation; other characters can not follow it. Example: \q=[END OF JOB]\0d\0a The above equation will output the string [END OF JOB] at the very end of a converted file. There is another use for the end-of-file code: to truncate a file when a specific character is encountered in the data stream. (This character may or may not occur at the true end of the file.) This is very useful in cases where some programs add an end-of-file code to any files they write, but you don't want them there. For instance, the DOS COPY command adds a hex 1A to the end of a combined file when you use the "+" operator to copy multiple files into one. To use this truncation feature, you first need to define a hex character that will be recognized specifically as the end-of-file character. At the beginning of your table, you must place the following setup code: \\Qxx ...where "xx" is a two-digit hex code representing the character that will trigger the end of file. For instance, the hex value 1A is usually recognized as the DOS end-of-file character, so the setup code -14- SNR+ VER 5.03 for that case would be \\Q1A. The next step is to add the \q equation to the table to actually process the end-of-file character: \q= This particular example will ignore the end-of-file character when it is encountered in the data stream, but will terminate processing of the file at that point, thus truncating the file. SPECIAL \s and \q INFORMATION Because of the way SNR assembles and categorizes conversion equations, any table that contains a \q or \s equation must be of a type that contains at least one 1:n or m:n equation (this is an equation type that has one or more characters on the left side of the equation, and two or more characters on the right side of the equation). \s and \q equations will not work in a table that contains only 1:0 or 1:1 equation types, because the 1:0 and 1:1 tables use a special high- speed array translation routine that bypasses the standard equation processing routines. A very simple way to coerce the table processor into categorizing a 1:1 table as an m:n type is to add the following equation to it: \80\80=\80\80 This will have no adverse effect on the data stream, since it converts two hex 80 characters as two hex 80 characters, but it will have the desired effect in SNR's table type categorizing. -15- SNR+ VER 5.03 WILD CARD CODES Now we get to the fun stuff. While SNR's string conversion capabilities are quite powerful on their own, there are inevitably times when it is necessary to perform conversions based on ambiguous patterns of characters, rather than on literal characters themselves. An example here would be if you had a comma-delimited ASCII file that was created by a database program, and you noticed that none of the phone numbers contained hyphens in them. It would be quite unfeasible to attempt to literally define all of the possible phone numbers without hyphens, and convert them into phone numbers with hyphens. You would end up with about 7 million equations that went something like this: 2220000=222-0000 2220001=222-0001 2220002=222-0002 ... 9989997=998-9997 9989998=998-9998 ...hopefully, you get the idea. To help prevent such exercises in futility, SNR provides a number of wild card codes than can be used in place of literal characters in conversion equations. The wild card codes allow you to search and replace on a set of characters rather than on a specific character. For instance, one of the wild card codes can be used to search for any of the numeric characters 0-9. Another can be used to search for any of the alphabetic characters A-Z. There are 12 wild card codes in all: \p any character \n numeric (0-9) \x full alphabetic (A-Z, a-z) \u upper case alphabetic (A-Z) \y lower case alphabetic (a-z) \m alphanumeric (A-Z, a-z, 0-9) \t punctuation \g non-whitespace \o whitespace (space, tab, return, line feed, vertical tab, form feed) \z ASCII-only (c < ASCII 128) \v printable (c > ASCII 31) \k non-printable (c < ASCII 32) The least restrictive of these, \p, will match any character that is encountered in the data stream. The most restrictive of these, \o, will match only the set of a few "whitespace" characters. The other wild card codes fall between these two extremes, and their associated character sets should be fairly straightforward to understand. On the search side of an equation, a wild card code can be used as a straight substitute wherever a literal or hex character would be used. For instance, if you wanted to delete any pattern of three numbers contained in parentheses, you would enter the following equation: -16- SNR+ VER 5.03 (\n\n\n)= This would delete such occurrences as (612), (000), (123) and so on. On the replacement side, however, wild card code usage requires some explanation -- here you can't simply use a wild card code in place of a literal character, as you can in the search string. SNR allows you to rearrange the order of wild card characters during conversion, so we need some method of numbering or otherwise identifying the position of each wild card code in the search string so it can be accessed in any order in the replacement string. This is done easily enough by convention: SNR sequentially numbers the characters in a search string. The leftmost search character is position 0, the next one is position 1, the next is position 2, and so on to the end of the string. The equation: (\n\n\n)= ...contains five characters: The left parenthesis is character 0, the wild card codes are characters 1, 2 and 3, and the right parenthesis is character 4. To use a wild card in a replacement string, then, we define the position of the character we wish to output, like this: (\n\n\n)=\p1\p2\p3 The \p code, when used in a replacement string, is always followed by the position number of the character you wish to output. In the above example, when three numbers enclosed in parentheses are encountered in the data stream, the parentheses will be thrown out and only the three numbers will remain (in the same order in which they were defined in the search string). If we change the equation to read like this: (\n\n\n)=\p3\p1\p2 ...the three numbers will be output in a different order. Let's return to the example we opened this section with -- that of replacing a series of phone numbers to add a hyphen between the third and fourth digits. Now that we have wild card codes available as a conversion tool, the task becomes much simpler: \n\n\n\n\n\n\n=\p0\p1\p2-\p3\p4\p5\p6 Look at the equation: seven numbers on the search side will be replaced by the first three numbers, followed by a hyphen, followed by the last four numbers. If the data stream contained 4315805, the equation would convert it to 431-5805. We could even handle area codes: \n\n\n\n\n\n\n\n\n\n=(\p0\p1\p2) \p3\p4\p5-\p6\p7\p8\p9 The equation as written defines 10 numbers on the search side to be replaced by: a left parenthesis, the first three numbers, a right parenthesis and a space, the next three numbers, a hyphen, and the -17- SNR+ VER 5.03 last four numbers. If the data stream contained 6124315805, the equation would convert it to (612) 431-5805. As an aid to equation readability, repeated numbers of wild card codes can use the repeating-character notation described in the first section: *(10)\n=(\p0\p1\p2) \p3\p4\p5-\p6\p7\p8\p9 This eliminates the need for you to specifically count how many \n codes you've entered in a search string. The above equation has exactly the same meaning as the previous equation. In a replacement string, you can also use the repeating-character notation, so long as you identify which position is being initially output: *(10)\n=(*(3)\p0) *(3)\p3-*(4)\p6 Look at the equation; it defines 10 numeric characters to be replaced by: a left parenthesis, three numeric characters starting at position 0, a right parenthsis, a space, three numeric characters starting at position 3, a hyphen, and four numeric characters starting at position 6. This is exactly the same equation as the one defined three paragraphs ago, but it is shorter and perhaps easier to read. Which notation you ultimately use -- long or short -- is purely a matter of personal taste and appropriateness for the equation at hand. Some equations will be more readable using the "long" notation, while others will be a natural fit for the "short" version. There is no need to dogmatically rely upon one or the other. MIXING WILD CARD CODES Wild card codes can be intermixed in an equation. For instance, the following equation will convert a comma, a space, two upper case letters, and five numbers into: a comma, a space, two upper case letters, two spaces, and five numbers: , \u\u\n\n\n\n\n=, \p2\p3 *(5)\p4 So if this string of characters was encountered in the data stream: , MN55044 ...the equation would convert it to: , MN 55044 CONVERTING SHIFT CASE Aside from \p, two other wild card codes can be used in a replacement string. Their purpose is to alter the shift case of an alphabetic character. The codes are \u and \y, for upper and lower conversion. As an example, we can make the sample equation dealing with the state and zip code used earlier a lot more universal with these new replacement wild cards: , \x\x\n\n\n\n\n=, \u2\u3 *(5)\p4 -18- SNR+ VER 5.03 This new equation will convert a comma, a space, two alphabetic characters of undetermined case, and five numbers into: a comma, a space, two upper case characters, two spaces, and five numbers. So if the following items were encountered in the data stream: , MN55044 , Ny10022 , il60301 , nJ07680 ...the equation would convert them to: , MN 55044 , NY 10022 , IL 60301 , NJ 07680 Here is set of equations that will convert a text file that is entirely in upper case into one that consists of upper and lower case, in sentence style: \u*00=\p0*01 \u*01=\y0 .=.*00 In this equation, when an upper case letter is encountered in the data stream, and flag 0 is OFF, it is output as-is and flag 0 is set ON. If an upper case letter is encountered and flag 0 is ON, it is converted to a lower case letter. If a period is encountered, it is output as-is and flag 0 is set OFF, so the next upper case letter will be output as-is. Any other characters in the file will be passed through as-is. From this example data stream: NOW IS THE TIME. FOR ALL GOOD MEN. TO COME TO THE AID. ...the equation would produce: Now is the time. For all good men. To come to the aid. How would you alter the above small table to convert a file from all upper case to initial upper case, where the first letter of each word is capitalized? Stated another way, what character appears before each word that could be used to control the flag? Here's the answer: \u*00=\p0*01 \u*01=\y0 .=.*00 \20=\20*00 Just by adding the last line to the equation, a space will set flag 0 to OFF, forcing the next character to be output as upper case. Now the previous sample data stream will be converted to: Now Is The Time. For All Good Men. To Come To The Aid. -19- SNR+ VER 5.03 WILD CARDS AND \s You can use wild cards in conjunction with the \s start-of-file code to remove a number of characters from the beginning of a file: \s*(256)\p= The above equation will remove the first 256 characters (of any type) from a file. BEWARE OF SIDE EFFECTS Wild card codes are powerful tools. They can also be powerfully troublesome if not used with care. Keep in mind that these wild card codes have a hierarchy of pattern-matching: they range from highly inclusive to highly restrictive. If you have two or more wild card equations in your table, be keen to the possibility that one of the equations might end up taking precedence over another, leading to unintended conversions. SNR bases its pattern-matching tests on the total length of search equations. That is, for any string in the data stream that could potentially match two or more table equations, the equation that is longest will be tried first. The next longest one will be next, and so on down the line until no further equations meet the pattern in the data stream item. So, for any two equations that begin with the same character, the longer one will be tested before the shorter one. If the longer one happens to match the data stream pattern, it is replaced immediately, and none of the other equations will be tested. If two wild card equations have the same search string length but begin with a different wild card code, the equation with the wild card code that is more restrictive will be tested first. That is, if an equation beginning with \x and one beginning with \v are both the same length, the one beginning with \x will be tested against the data stream first, because the set of alphabetic characters is more exclusive than the set of all printable characters. Design your wild card search strings carefully to take this behavior into consideration, and your conversions will be more successful. -20- SNR+ VER 5.03 VARIABLE-LENGTH CONVERSIONS Up to now, the conversions that we have been discussing have dealt with fixed-length patterns of characters: some number of characters in a search string gets converted to some other number of characters in a replacement string, and we have been able to specifically count the number of characters on either side of the equation. Now we are going to explore the method SNR has of defining repeating characters or wild card codes whose actual number is unknown. These types of equations are called "variable-length" conversions. Variable-length conversions are used when you want to search on some identifiable set of characters -- maybe a line of text, maybe a phrase, maybe a command parameter -- but you don't know HOW MANY characters will be involved in the actual pattern. For instance, a line of text might contain one word, or it might contain fifty words. An imbedded text command might contain one parameter, or it might contain twelve parameters. A comma-delimited database field might contain ten characters, or it might contain twenty-five characters. The one thing that these types of text patterns -- a line, a parameter, a field -- have in common is that we wish to treat each of them as a unit, even though we don't know how many characters they will contain. Using variable-length conversions, you can define these higher-level units of data and translate them, throw them away, or rearrange them as needed. SEARCH STRING FUNDAMENTALS A variable-length string is declared to be a set of a specific type of character (literal or wild card) that matches from zero to a stated maximum length of similar characters in the data stream. Since variable-length search strings are basically unspecified lengths of similar characters, we can draw upon the notation that we prescribed earlier for the repeating-character strings: *(40)\20 The above portion of a search string should be familiar to you as the definition for 40 spaces. To turn this into a variable-length definition, we add the \^ variable-length code: \^*(40)\20 This search string fragment now defines "up to" 40 spaces. Likewise, the following fragment defining a string of 25 alphabetic characters: *(25)\x ...can be redescribed in variable-length terms like this: \^*(25)\x -21- SNR+ VER 5.03 This search fragment now defines a string of "up to" 25 alphabetic characters. It is important to adhere to the following rules when you use variable-length search strings: - An equation cannot begin with a variable-length search string. - Each variable-length search string must be followed by a fixed- length terminator string. These restrictions are not quite so arbitrary as they may seem. As for the first rule, each equation must have some character or set of characters of fixed length that can begin the search pattern. The variable-length string can come right after this. As for the second rule, there must be something to tell SNR when the end of a variable- length string has been met; some specific character or set of characters must be defined that can act as a terminator to the variable-length string. The second rule also implies that the search side of an equation cannot end with a variable-length string. So here is a complete example of a valid variable-length search string: \v\^*(80)\v\0d\0a= Look at the equation. It defines: a printable character followed by up to 80 more printable characters terminated by a carriage return/line feed, to be thrown away. The effect of this equation will be to throw away full lines that are less than 82 characters long, or to throw away the last 82 characters of a longer line. Multiple variable-length strings can appear in an equation: "\^*(30)\v","\^*(30)\v","\^*(30)\v"\0d\0a= The above equation will search for: a quote followed by up to 30 printable characters followed by a quote-comma-quote terminator followed by up to 30 more printable characters followed by another quote-comma-quote terminator followed by up to 30 more printable characters terminated by a quote and carriage return/line feed, and throw it all away. You may recognize this particular pattern as being characteristic of the comma-delimited ASCII file that many database and mail-merge programs support. The two examples we used above employed literal characters as terminator strings. It is equally valid to use wild card codes as terminator strings: \n\^*(20)\n\t= The above equation defines a number followed by up to 20 more numbers followed by a punctuation character to be thrown away. When you use wild card codes as a terminator to a variable-length string which itself uses wild cards, remember the earlier admonition concerning the hierarchy of wild card pattern matching. It applies -22- SNR+ VER 5.03 equally here. Specifically, a wild card terminator must be more restrictive in type than the variable-length wild card string it is terminating. For instance, the following variable-length string is invalid: \n\^*(10)\n\p= The \p code used as the terminator has a higher inclusion value than the \n code. The result will be that this equation will generate incorrect output, if indeed it outputs anything at all. REPLACEMENT STRING FUNDAMENTALS The prior discussion concerned itself exclusively with the procedures for defining the search side of an equation. Now we will turn our attention to defining the replacement side of a variable-length equation. Since we don't know the actual number of characters that a variable- length string will pick up, it is not possible to use the \p notation to identify precise character positions for output. Instead, we identify groups of variable-length strings by the positions of the group: the first variable-length string on the search side is number one, the next such string in the equation is number two, and so on. (This is in contrast to the \p code, which begins at zero.) Up to 40 variable-length strings may be entered in a single equation. The variable-length replacement strings are defined as \^1, \^2, \^3 and so on. Let's take an example based on the comma-delimited ASCII file, used earlier: "\^*(30)\v","\^*(30)\v","\^*(30)\v"\0d\0a=\^1\09\^2\09\^3\0d\0a While the search side of the equation remains the same as before, we have now introduced the replacement side. The replacement will be the first variable-length string followed by a tab code (the hex 09) followed by the second variable-length string followed by a tab code followed by the third variable-length string followed by a carriage return/line feed. The effect of this equation will be to convert a comma-quote delimited file into a tab-delimited file. (Digression: for purposes of illustration this example is fine, but is there a better way to accomplish the same result? The answer appears in a later section.) You may have observed by insight that it is possible to rearrange the strings by changing the order in which they appear on the replacement side of the equation. The procedure is straightforward and obvious, so I won't bother with an illustration. How do you output a wild card character that has been defined as a terminator? Since you can't pick the character out of the data stream by exact position, SNR provides a notation that describes a terminator string. Since each variable-length string in an equation is numbered from 1 to 40, and by definition each variable-length string must be followed by a terminator string, it makes sense that the terminator strings are also numbered from 1 to 40. -23- SNR+ VER 5.03 The replacement notation for terminator strings is \^:1, \^:2, \^:3 and so on (the colon being the only difference between them and the variable-length string identifier). Now having some way of differentiating between variable-length strings and terminators, we can convert equations such as the following: \u\^*(15)\y\u\^*(15)\y\t=\p0\^1\^:1\20\^2\^:2 This admittedly contrived example serves to illustrate the placement and usage of the various codes. The search side defines an upper case character followed by up to 15 lower case characters terminated by an upper case character followed by up to 15 more lower case characters terminated by a punctuation character. The replacement side outputs the first character of the string (the first \u on the search side) followed by the first variable-length string followed by the first terminator followed by a space followed by the second variable-length string followed by the second terminator. For a concrete example, assume that this string occurs in the data stream: SuperDuper! UnquestionablyAwesome. Our example equation will convert it to this: Super Duper! Unquestionably Awesome. Let's try switching the order of a few elements in the equation: \u\^*(15)\y\u\^*(15)\y\t=\^:1\^2\20\p0\^1\^:2 The search side remains the same; the replacement side has changed. Study it and see if you can follow the meaning. Perhaps this will help: given the same example data stream as above, the altered equation will produce this output: Duper Super! Awesome Unquestionably. PADDING VARIABLE-LENGTH STRINGS A specialized use of variable-length replacement strings allows the strings to be padded out to their maximum width. Normally, of course, the strings are output to their "natural" width -- however many characters are contained in the data stream are what gets output during replacement. Padding the strings adds blank spaces to the left or right of the text proper, and has the effect of aligning the text within a field that is as wide as the maximum width specified in the variable-length search string. To invoke variable-length padding, the alignment codes R and L are inserted before the variable-length string number in a replacement string. For instance, \^L1 will left-align the first variable-length string; \^R3 would right-align the third variable-length string. Here's how the codes are used in an equation: -24- SNR+ VER 5.03 "\^*(30)\v","\^*(30)\v","\^*(30)\v"\0d\0a=\^L1\^L2\^R3\0d\0a Taking our well-worn comma-delimited example yet again, pay particular attention to the replacement side. Those codes specify that the first variable-length string will be output left-aligned to a width of 30 characters, the second variable-length field will be output left- aligned to a width of 30 characters, and the third variable-length field will be output right-aligned to a width of 30 characters followed by a carriage return/line feed. What we have done in this example is converted a file containing comma-delimited fields into a file containing fixed-length fields. The widths of the fields can be changed simply by changing the width values in the search strings: "\^*(35)\v","\^*(25)\v","\^*(40)\v"\0d\0a=\^L1\^L2\^R3\0d\0a In this case, the replacement side of the equation is the same as before, but the search side now contains different widths for each of the fields. This will affect the appearance of the output when the text is padded; it will not affect the appearance of the text if the alignment codes are removed. PAD CHARACTER By default, variable-length strings are padded with spaces when the R or L alignment codes are specified. You can select a different padding character by placing the setup code \\Axx at the beginning of your table, where "xx" is a hex value representing the character you wish to use as the padding character. Only one padding character may be defined per table, and once selected, it can not be changed. As an example, you could pad your variable-length text strings with periods by using the setup code: \\A2E -25- SNR+ VER 5.03 HINTS AND TIPS The conversion capabilities of SNR are very sophisticated. Some of them are subtle, and require some study and experimentation to master. Here is a list of a few pointers that might help you avoid the pitfalls of inexperience. 1. Lots of smaller tables are better than one large table. It is sometimes tempting to create a huge, do-it-all conversion table that takes care of your conversion task all at once. Let me tell you from experience that it is much better to break down your conversion into a series of smaller tables that are run one after the other. One reason behind this is that it is much simpler to pinpoint problems when you take your conversion a little bit at a time, and you won't waste your energy poking around a large table feebly trying to discover why your text isn't converting the way you thought it should. Remember that SNR allows you to run up to 20 tables at one time on the command line anyway, giving you the same benefit as having one large table. Another reason to break your task into smaller pieces is that some of the more advanced conversion models -- such as variable-length strings -- may have unwanted side effects on other equations or vice versa. It is safer to isolate the complex portions of a conversion task into their own tables to insure that harmful side effects are not introduced. 2. Use literal characters over wild cards wherever possible. Wild cards are very powerful, but they require much more time to process than literal characters. Whenever you can, it is preferable to define conversion equations using literal characters instead of wild cards. If your conversion task can be represented by a few well- defined data patterns, it's a good candidate for literal equations. If, on the other hand, you have a particular need to search a wide range of data patterns, then the wild card equation is your best choice. Remember the question I posed back in the variable-length section? I asked if there was a better way than using variable- length strings to convert a comma-delimited text file into a tab- delimited text file. The answer is yes. If you think about it a little bit, there is no real need to do any conversion of the data within the delimiters at all. Only the delimiters themselves need to be converted. So rather than chewing up a lot of CPU cycles scanning the data stream for variable-length text, the following two equations will perform the conversion much faster: "= ","=\09 An added benefit to using this table over the other one is that this version does not need to be modified when the number of fields changes, whereas the other one does. I guess this is one of those situations where Less Is More. -26- SNR+ VER 5.03 3. Use context flags rather than variable-length patterns when you are seeking to compress the data stream. This is another issue related to efficiency. Variable-length conversion strings provide a means of reading an unspecified number of consecutive characters and eliminating them, in effect compressing the data stream. But equations using context flags such as *ic or *ig provide a much faster conversion than variable-length equations. 4. Debug your tables an equation at a time. Debugging is a cross between an art and a science; it requires both discipline and inspiration to do it well. The first step in effective debugging is to find the trouble spot and focus your attention on it. The easiest way to do this in SNR is to place a comment code (backslash followed by a space) in front of every equation in your table, and enable each equation one at a time. Test a sample of your data file for each new equation that you enable. Eventually, you will hit upon the one equation that is fouling things up, and you will be able to alter it or isolate it by moving it to another table, where it can be dissected and analyzed apart from other equations. Once the trouble equation has been found, look at it carefully. Consider these questions: - Are you sure it is defined according to the rules set forth in this document? - Does each element of the equation logically match a likely pattern in the data stream? - Are your wild card definitions too vague to match a specific pattern in the data stream? - Are there any possible conflicts between elements of the equation that would render it invalid? - Can the equation be restated equivalently in different terms, that is, using a different conversion model (for example, transforming wild cards into literals)? - Is what you are trying to do reasonable or even possible in the SNR conversion paradigm? - Have you done something like this before and made it work? - Is there a similar equation shown in the sample tables? - Could there be a bug in SNR? Seeking the answers to these questions will help guide your troubleshooting steps in a logical, scientific manner. 5. Use lots of comments. This point was brought up in the first section of the manual, but it bears repeating. 'Nuff said. 6. Peruse the provided example tables. The range of possible uses of SNR, and the examples to support them, are beyond the scope of this manual. Many of the more complex and subtle uses of conversion strings can only be -27- SNR+ VER 5.03 appreciated by experimentation or by looking through the example tables provided in the archive file. The tables may help provide you with hints on syntax and usage that this manual was unable to. If you register the program, you will be entitled to telephone or on-line support (toll at your expense) and I will be happy to help you solve your conversion problems. -28- SNR+ VER 5.03 MISCELLANEOUS BIT STRIPPING In prior versions of SNR, the default action was to map all incoming characters to 7-bit ASCII before running them through the equations. In version 5, this has been changed: no "bit-stripping" is performed by default. If you wish to have your table map incoming characters to a 7-bit ASCII set, you must enter the \\L7 setup code at the beginning of your table. NOTES The program will run on any IBM-compatible computer using DOS 3.0 or higher, with a minimum of 512K RAM. SNR was written in Borland C++ 3.1. DISCLAIMER This program is distributed as shareware. Use it, copy it, upload it, give it to your friends. Please distribute only the complete program in archived form, including all document and sample files. No warranties, either expressed or implied, are given by the author or distributor of the program, and the user accepts all risk of damage arising out of the application and use of the program. REGISTRATION If you like SNR, please register your copy for $50. At that price, it's one of the bargains of the text processing world, and I'll bet you recoup that investment dozens, maybe hundreds, of times over in the work it will save you. Registered users will receive a version of the program that does not have the compliance delay screen. Also included FREE with the registered version is TurboEdit, a DOS text editor that provides a multiple-window, mouse-supported editing environment with a built-in ASCII table and calculator (very useful for editing your conversion tables). Please fill out and return with your payment the form in the file ORDER.FRM. Registered users are guaranteed a response to technical support questions by phone, mail, or CompuServe. Unregistered users will receive responses on a time-available basis, but I can't guarantee that I will respond to those requests. -29- SNR+ VER 5.03